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Abstract 

A central premise of the discovery-learning and progressive education movements was that 
the child's own questions are the most appropriate starting point for instruction. Recent advances 
present new opportimities for discovery-oriented learning. We have been attempting to create a 
classroom environment which afford students the opportunity to pose meaningful problems within 
the domain of a specific scientific challenge. One component of this environment is a brief video 
tape which invites students to generate problems which would have to be solved in order to carry 
out a mission to the planet Mars. Two studies explored several aspects of sixth graders* 
performance in this environment In the first study, we categorized the content of the problems 
which students posed and attempted to evaluate changes in problems across three stages of the 
process. In the second study we compared the effectiveness of our video with a typical educational 
video about travelling to Mars. While these studies confirmed that these students arc able to pose 
"educationally worthwhile" problems in in this environment, we concluded that morc explicit 
guidance is needed. Proposals for providing this guidance and additional research questions are 
described 
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Ejects of Generative Video on Students* Scientific Problem Posing 

The National Space Grant College and Education Fellowship Act of 1987 gave NASA the 
mandate to greatly expand its educational mission. In order to "help maintain America's 
preeminence in aerospace science and technology/ the act mandated that NASA devote a portion 
of its appropriation to ensuring a trained pool of scientist and engineers in the years to come. 
While a substantial portion of the Space Grant resources are devoted to University-level training, 
K-12 education is also an important focus. One of the primary objectives specified for individual 
Space Grant programs is: 

To arouse the interest of a generation of K-12 students in mathematics and science, to 
improve their levels of competency in such subjects, and to stimulate their collective 
interests in, preparation for, and dedication to careers in diverse technology based 
occupations (NASA, 1990, p, 10). 

This statement embodies the two strongly interrelated challenges facing NASA and the nation as a 
whole. The first of these challenges is fostering an interest in science careers in general and the 
aerospace sciences in particular. Interest among freshmen in science and engineering over the last 
decade has dropped by one quarter to one third, depending on the field, (NASA, 1990). In many 
countries, including die U.S., researchers find a substantial decline in interest in math and science 
during the high school years (Jones, 1988; Lehrke, Hoffman, & Gardner, 1985). 

The second challenge facing NASA is developing in today's youngsters the academic 
abilities need to prepare for college-level science and engineering careers should they choose them. 
This academic challenge can be further considered in tenns of two interrelated components. The 
first is in basic skills, i.e., the verbal and math skills traditionally measured by the SAT and other 
standardized tests. The second component of the academic challenge is in the area of critical 
thinking skills. The poor problem solving abilities of U.S. students is well documented. 

O 
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Particularly in complex academic domains such as math and science, students demonstrate 
marginal (and declining) ability to generate and evaluate arguments, engage in informal reasoning, 
and generate and solve moderately complex problems. (Jones, 1988; Kouba, Brown, Lundquist, 
& Silver, 1988). While the basic skills of students who plan to purse college majors in math and 
sciences has remained relatively constant, their ability to to solve more complex problems 
continues to decline; Among the students who do not plan to study science and math in college, 
the decline in basic math and science skills as well as critical thinking skills has been substantial 
(Jones, 1988). While a great deal of research has been devoted to improving academic abilities, 
little formal research has been directed at increasing interest While NASA and other federal 
agencies have invested extensively in K-12 outreach programs designed to increase interest, we 
lack an adequate research base for evaluating the impact of these programs on students academic 
achievement 

With the support of the Tennessee Space Grant Consortium, we have engaged in a research 
and development program designed to increase student interest in math, science and technology in 
a manner that dixectiy impacts student academic achievement We are attempting to create a 
classroom learning enviroiunent which anchors K-12 math, science, and cross-curricular content 
within the context of interplanetary travel-more specifically, planning a human mission to Mars. 
We are following a generative, problem-solving approach based on current psychological and 
educational research and are attempting to incorporate emerging multimedia technologies. A central 
feature of our approach is that it contextualizes learning around solving problems that the students 
themsehes generated 

Our efforts are rooted in the writings of Dewey, and the British "open learning" and the 
American "discovery learning" movements. Central to these approaches is the belief that a 
student's own questions and wonderments represent a discrepancy in knowledge and a desire to 
learn, and are thus the most appropriate starting point for instruction. Scardamalia & Bereiter 
(1990) describe the historical antecedents of this perspective on student inquiry: 
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Starting with the work of Susan and Nathan Isaccs (1930), the "chUd's own question" has 
been seen as a prized object that should be at the center of the curriculum. What makes a 
question the "child's own" is that it springs from a deep-seated mterest of the child or arises 
from an effort to make sense of the world.... The source of questions is a gap or 
discrepancy in the child's knowledge or a desire to extend knowledge in some direction (p. 
3) 

The 1960's saw a massive investment by the National Science Foundation in developing 
and disseminating inquiry-oriented science curricula designed to present learners with the 
opportunity to discover important scientific principles by interacting with everyday objects. These 
programs were explicidy based on Piaget's theory of developmental sequences which presumed 
that experimentation strategies emerge during particular periods of the child's development Thus 
the early discovery learning environments were designed to facilitate the emergence of scientific 
skills rather than teach them direcdy. One popular curriculum, the Science Curriculum 
Improvement Study (SCIS) followed a three-stage cycle consisting of exploration, concept 
introduction, and concept application (/^itkin & Karplus, 1962), Rakow (1986) described the 
exploration phase as 

In the exploration stage (sometimes referred to as "messing around") students are given 
unfamiliar materials (for example, a set of geoblocks or a set of batteries, bulbs, and wires) 
and are instructed to find out everything that they can about the materials. This 
unstructiu^d exploration leads students to ask further questions about the materials, which 
they may wish to investigate (p. 21) 
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In the concept introduction phase, the class uses the "concept invention" strategy to discover a 
scientific concept based on observations made during the exploration phase. In the concept 
application phase, the class applies the newly established scientific concept to a new situation. 
Many variants emerged and later programs typically incorporated more guidance, but all reflected 
the presumption that scientific strategies were something that "emerged." 

Discovery learning ajlvocates were able to amass substantial evidence that discovery 
learning programs led to greater performance on measures of both traditional basic science concept 
knowledge and "scientific process" skills (see Bredderman's 1983 meta-analysis). However, 
substantial controversy surrounded the large-scale implementation of discovery learning methods. 
Teachers' observations confirmed their commonsense notion that students following such 
approach will rarely "discover" a significant number of meanmgful scientific concepts in such a 
setting. Teachers also observed that the confusion and frustration that can result from unguided 
inquiry often eliminates the increased motivation thought to follow when students are able to 
pursue their own questions (Linn, 1986). Despite the empirical support for the approach and the 
massive investment in dissemination and teacher training, the 1970's saw a return to the more 
traditional direct instructional approaches which arc rooted in a "transmission/reception" model of 
teaching and learning. While these highly structured approaches offer clear benefits in terms of 
classroom management and assessment, there is broad agreement among researchers that such 
approaches are at odds with contemporary psychological theory (e.g., Resnick, 1989) and stated 
educational goals (e.g., AAAS, 1989). This renewed reliance on direct-instruction approaches has 
led science educators and educational researchers to reconsider the inquiry-oriented approaches of 
the previous generation. 

Recent advances in cognitive and instructional psychology and the increased availability of 
classroom computing, multimedia, and telecommunications technology present new opportunities 
for discovery-oriented learning (Blumenfeld et al., 1991). Earlier approaches to discovery- 
learning were based on theories of cognition, motivation, and learning which focussed on domain- 
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general processes. Following the general paradigm-shift in cognitive psychology, researchers 
have begun reconsidering discovery learning witii tiie assumption that cognitive processes and 
motivation are strongly bound to specific domains, or even specific content within a domain. In 
practice this has led to a focus on teaching generalizable cognitive processes in the context of 
meaningful, real-world content Recent research carried out in the area of "Cognitive 
Apprenticeship" (Collins, Brown, Sl Newman, 1989), Anchored Instruction" (Cognition and 
Technology Group at Vanderbilt, 1990, 1991, 1992, in press), and constructivism (e.g., Cobb, 
Wood, & Yackel, 1992, Duffy & Jonassen, 1992) provide a useful framework for reexamining 
and improving inquiry-oriented approaches. In general, these new approaches situate inquiry 
within a pre-specified problem" space," and attempt to induce transfer by leading students to apply 
newly-learned generalizable skills within a new domain. 
The Mars Challenge 

Our prototype learning environment is designed to lead students to pose educationally 
worthwhile problems and then support student inquiry into solving these problems. Figure 1 
illustrates the three basic levels of the conceptual model v hich we have applied to the Mars 
Challenge project, and lists the instructional activities, learning affordances, and materials 
associated with each. The first level. Problem Generation, focuses on students' posing, defining, 
and categorizing of problems within the problem space. The second level. Knowledge Distribution 
& Teamwork, is concerned with establishing a cooperative environment based on individual efforts 
directed towards the larger group's common goal. The third level. Using Knowledge Tools, is 
concerned with facilitating the use of educational and reference resources to solve selected 
problems. While these levels are roughly sequential, the levels are highly iterative and 
interdependent In particular, the activities that make up Uie problem generation level arc viewed as 
ongoing throughout the entire process. 

The focus of our efforts to date and the primary topic of this paper is problem generation. 
[Additional details regarding the larger learning environment are described in Hickey, Petrosino, 
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Pellegrino, Goldman, Bransford, & Sherwood (1992).] We consider problem generation most 
essential because it affords the interest in the general topic and specific problems needed to 
motivate students during subsequent self-directed inquiry and learning. 

In order to introduce students to the topic and the problem generation task, we developed a 
short video using existing NASA footage. The Mars Mission Challenge video suggests the range 
of factors that would have to be considered and challenges students to generate questions that 
would have to be asked and problems that would have to be solved to carry out such a mission. 
Over a series of classroom trials, problem generation has evolved into a three-stage process (see 
Figure 1) In the problem posing phase, students view the video and pose as many problems as 
they can. In the problem definition phase, students further define and prioritize their problems; in 
the problem categorization phase, students sort and categorize their problems. 

In order to more formally evaluate the problem generation process, we must first establish 
criteria for evaluating problem generation outcomes. As the "artifact" of the process (Blumenfeld 
et al., 1991) the problems themselves are the most obvious outcome. Our primary goal is to lead 
students to generate "educationally worthwhile" problems. We define these as problems which 1) 
fall within the general problem space of planning a mission, 2) are sufficiently open-ended to 
support inquiry in a manner that would require students to defend their solution, and 3) can 
plausibly anchor student inquiry in a manner that supports learning academic content In the two 
studies reported here we have attempted to further define the first two of these criteria. 

Our initial problem generation environment was minimally guided. While we did establish 
a detailed activity framework, this was necessary to guide youngsters through a task that would be 
challenging even for adults. Our objective was to provide a minimal amount of intellectual 
guidance within each step of the activity, and allow our classroom experience and ongoing 
theoretical development to shape the inclusion of additional guidance. We reasoned that excessive 
focus on teaching students about generating "good" problems would detract attention and interest 
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from the content To the degree possible, we allowed the students' knowledge of the content to 
guide the generation process. 

Study One 

Our pilot studies showed us that the difficulty which students (and teachers) have with 
domain-specific problem posing stems more from unfamiliarity with the task rather than inherent 
inability. Following some initial confusion over what exactly was expected of them, nearly all 
students were able to pose questions which could form the core of important and meaningful 
problems. The objective of this first study was to evaluate the content and quality of the problems 
which students pose and evaluate changes in the set of problems across the three phases of the 
generation process. 
Method 

Subjects. Forty-eight students from two sixth-grade classrooms participated. The first class 
consisted of average/high ability students from a suburban school. A few of these students had 
participated in space-ielated activities, including "Space Camp." The second class consisted of 
low/average ability students a from racially mixed inner-city school Changes in procedure for the two 
classes precluded any analysis of the differences between the problems posed by the two groups. 

Procedure. Students in the first class were were divided into four groups of six or seven 
students. After viewing the video as a class, the groups were instructed to "think up as many good 
problems as you can." Each group then took turns reviewing the videodisc scene-by-scene while 
attempting to pose additional problems. Then students participated in a pilot problem categorization 
activity for which we did not collect formal data* 

Students in the second class viewed the video and were then instructed to individually pose as 
many problems as diey could. They were then placed in seven groups of three or four and instructed to 
generate a master set from each individual's list, recording each problem on a single slip of paper. The 
groups together viewed the video a second time. The groups were then instructed to sort problems into 
categories and to come up with a name for each category. Each group then placed all similarly 
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categorize problems in an envelope and wrote the name of the category on the envelope* With the first 
author's guidance, the entire class then categorized each groups* set of problems* This resulied in five 
sets of problems, each consisting of the envelopes of problems from the individual groups. Students 
then reassembled into five groups, with each student joining the category of choice. These new groups 
were instructed to assemble a master list of "the most important" problems ft-om all of the individual 
problems which they had received* 
Res\iit.s and CQn<?IysiQPS 

Problem posing . The four groups of students in the first class and the seven initial groups of 
students in the second class produced 346 problems. The groups in the first class posed 50, 56, 28, 
and 48 problems, while the groups in the second class produced from 1 1 to 37 problems, with an 
average of 23.4. Twenty-seven problems were eliminated because they were unreadable or 
incomprehensible. Many of the problems students posed were poorly defined reflections of a general 
question (e.g., "What about food?") while some were the well defined problems we had been seeking 
(e.g., "How many pounds of food will we have to bring?"). Several attempts to define reliable (and 
valid) measures for evaluating problem quality failed. The difficulty in defining criterion stemmed from 
1) the generally poor definition of the problems, 2) the limited theoretical framework regarding the 
generation process, and 3) the absence of a clear framework for incorporating the problems into 
subsequent leanung activity. 

To siunmarize the content and nature of the problems, the first author evaluated the problems in 
light of their anticipated role a<5 anchors for in«;truction in subsequent learning activities. Based on the 
materials available and the need to delimit the curriculum, we had initially decided that it would be 
reasonable to support inquiry in several broad content areas (Hardware, Mission Configuration, Health, 
Life Support, and Surface Activity; we later added politics)* We considered a problem "inside" the 
problem space if it explicitly or impliciUy reference content determined to fall under one of these six 
content areas* A great deal of inference was involved in assigning individual problems to particular 
content areas. Thus the analysis of problem content is exploratory and not intended intended to test any 
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formal hypothesis. Table 1 summarizes the content of the 319 problems presented by each of these 
groups. We found that 188, or 59% of the problems were categorized as being inside the problem 
space. Most of the remaining problems are conceivably relevant, but which we have chosen to exclude. 
The category "conditions on Mars" was not considered as inside the space because these items were 
merely questions about Mars but were not considered in the context of how they impact on a mission. 
For example, questions about the surface of the Mars would have been considered inside if they had 
been posed within the context of planning a landing site. While problems relating to colonization or life 
on Mars ate certainly relevant to planning a mission, we deemed them outside the domain of 
manageable inquiry. Clearly, addressing problems on the boundaries of the anticipated problem space 
represents a major challenge. While we have chosen not to support further investigations into these 
topics, we will not necessarily force students to abandon them either. The category Problems appears 
to be the result of some students interpreting "problems" as "things that might go wrong" rather than as 
a challenge needing to be solved, suggesting a need for more explicit instructions. Only 18 problems 
were deemed frivolous. 

An important concern is the coverage of topics within groups of students. While we want 
students to eventually focus on more specific topics, we were concerned that students might focus their 
efforts on a single topic from the start, rather than allowing a focus to emerge within the larger problem 
context. In the fnrst class, three groups posed problems from all five major topic categories (Politics 
was excluded), while one group posed problems from just three categories. In the second class, where 
the format appeared to lead the groups to pose fewer problems, only one group posed problems from 
all five content categories; two groups posed problems from four of the categories, two groups from 
three categories, and two groups posed problems from just two of the categories. The marginal 
coverage in the second class suggests appears to favor having students engage in the initial problem 
posing activity in small groups (as in the first class), rather than as individuals. 

Problem categorization. As a pilot test of the categorization process, we were quite pleased with 
the process and outcome in the second class. The students engaged in spirited discussions about the 
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quality and categorization of each others problems. We were particularly encouraged because students 
appeared to be emotionally invested m the problems which they and their groups produced We also 
observed students spontaneously consulting existing classroom reference material (the encyclopedia) to 
support their argiunents, and the students reported enjoying the process. 

Table 2 lists the five categories and the "most important" problems which emerged from the 
categorization process in the second class. While the categories are fairly coherent, the problems are 
not very well defined and the relationship between problems within categories is not always apparent 
While there is considerable overiap in the categories, this is not necessarily undesirable, since we would 
expect different groups to consider a similar topic from different perspectives. 

The materials used in the second classes' generation activity were color-coded, providing a 
record of interim products at the individual posing, group definition, and group categorization stages. 
Evaluation of the problems collapsed across students at each of the three phases showed the proportion 
of problems scored as inside was roughly constant (around 60%). The questionable reliability and 
validity of our measures precluded formal evaluation of changes in the problems across the phases, and 
our cursory examination revealed no evidence to suggest that the problems were improved any way. 

In summary, we conclude that students at this level need more explicit guidance during problem 
generation activity of this sort, but that we still lack the theoretical guidance and empirical findings to 
direct our inclusion of this guidance. It is likely that limitations in domain knowledge and general 
reasoning ability, as well as insufficient comprehension of the task are all factors in student 
performance. Thus it is not clear whether the most appropriate course of action is to provide students 
with more domain knowledge, more general activity guidance, or both. We expect continued 
theoretical development and classroom experience to help answer these questions. 

Study Two 

The first study demonstrated that while the problem generation task which we had defined 
was effective at leading students to generate educationally worthwhile problems within a specified 
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domain, there is much room for improvement. A shortcoming of the first study was the lack of 
reliable and valid problem quality measures. In Study Two we attempted to evaluate a very 
specific aspect of the problem generation environment, the Mars Challenge Video, while attempting 
to use a new measure of problem quality. 

The Mars Mission Challenge video represents a departure from the format of typical 
educational videos. Following from the traditional transmission/reception model of the teaching 
and learning process, most educational videos are designed to present a specified body of 
information. For example, other NASA Mars mission videos which we reviewed present facts 
about the planet, describe one or more potential mission scenarios, and describe how NASA plans 
to solve the many challenges that the mission presents. Following from a generative model of 
learning, the Mars Mission Challenge was designed to prompt students to construct questions to be 
asked and problems to be solved in addressing the challenge, based on each students' prior 
knowledge and interests. 

In a study motivated by questions similar to ours, Scardamalia and Bereiter (1990; see also 
Scardamalia & Bereiter, 1991) compared the questions generated by students in CSILE 
(Computer-Supported Intentional Learning Environment) classes following a unit of instruction on 
a particular topic (endangered species) with the quCotions generated by CSBLE students who had 
received no instruction on the topic and were challenged to "write questions reflecting what they 
wondered about or needed to know to advance their understanding of endangered species" In the 
former condition (referred to as "text-based" questioning) smdents tended to ask specific factual 
questions which could be answered by information presented explicitly in the lesson text; in the 
latter condition (referred to as "knowledge-based" questioning) students asked questions based on 
their prior knowledge. Analysis of the questions revealed that the knowledge-based questions 1) 
represented a higher-level of mquiry, 2) required a more complex answer, 3) were rated as "more 
interesting" by teachers, and 4) required more complex search for answers. 
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This study compares the problems posed by students who viewed the Mars Mission 
Challenge video with problems generated by students who viewed a comparison Mars mission 
video. We expected that the comparison video would lead students to pose more specific problems 
about facts presented in the video, and that their problems would be similar to the questions 
generated in the *'text-based" condition above. In contrast, we expected the Mars Challenge video 
to lead students to pose problems based more on prior knowledge, and that their problems would 
be more similar to the questions generated in the "knowledge-based" condition above. Specifically 
, we expected the comparison students to pose more low-level questions that would call for yes/no 
or factual answers, whereas we expected the Mars Challenge video to lead students to pose more 
high-level questions that we expect will be more useful anchors for subsequent inquiry. 

This study was also used to pilot test a new approach to the problem generation task. We 
felt that the goals and motivation we provided for the students to pose, define, and categorize 
problems in the first study were rather abstract and not connected to any real-world activity. Thus 
we attempted to situate the generation activity around the eventuality of asking different kinds of 
science experts for help in figuring out how to solve selected problems. We modified the 
instructions and the materials to situate the the need to posed and define good problems in the need 
to make the best use of the experts' time. The need to categorize problems was situated in the goal 
of asking the most 2^)propriate question to the different kinds of experts. 

Students . Twenty-nine sixth graders from a suburban parochial school participated. The 
teacher assigned students to either the generative video or informative video condition using 
stratified random assignment She was specifically asked to distribute the highest ability, lowest 
ability, and high domain-interest students evenly among the two groups. 

Materials . The Mars Mission Challenge is a seven minute video which presents a myriad 
of images relative to planning a mission. It is set to upbeat music and features a narration that 
focuses on the many problems that would have to be solved in planning a mission. While the 
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images present many clues to the students, we were very general in order to avoid pre-empting 
students. The video narration explicitly challenges students to pose problem related to the Mars 
mission and states that "each image has meaning and some scenes have many meanings." The 
video was designed so that after the first viewing, it can be reviewed scene-by-scene to further cue 
additional problems and further define problems already posed. 

The comparison treatment featured a six-minute long video that presented a hypothetical 
mission to Mars. The narration describes the various aspects of the proposed mission in a fashion 
that would lead a young viewer to believe that NASA is already committed to such a mission 
configuration and is planning to actually carry it out Some of the images on the video are the same 
as the images used in the Mars Challenge Video. In terras of content, the comparison video gave 
details about extended activity and short-term colonization of the planet, whereas the Mars 
Challenge video presented surface activity in the context of a short-term visit Additionally, the 
Mars Challenge video raised the issue of past or present life on Mars, whereas the comparison 
video made no such reference. The source copy of the video was close captioned, so we re- 
recorded the tape with a black bar across the portion of the bottom of the screen where the closed- 
caption had been. 

We created new materials to accommodate the changes in our approach. This included 
illustrated vignettes of nine different NASA scientists which featured a 3-5 sentence paragraph 
describing the experts and their domain of expertise. We also created a form for individual 
students to assign each of their problems to one or more of the experts. At the bottom of this form 
was a list of the nine experts and instructions to pick a first, second, and third choice of expert 
which the student would prefer to work with. 

We also assembled nine small packets of NASA educational materials and a National 
Geographic article (One for each expertise area) for students to search through for information 
which was relevant to their problem in the final phase of the ac>ivity. 
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Procedure. The entire class was given a brief introduction and told that we would be 
working with them for several more days in their classroom. All students were read the same 
instructions (Appendix A) and given a piece of paper which staled at th« top: 

Can you think of any important questions that would have to be answered and problems 
that would have to be solved to send women and men to Mars and bring them back home 
safely? Please write down all of the questions and problems which you can think of 

The two groups went to opposite ends of the room and watched the program on video monitors. 
Students then returned to their seats and were instructed to begin listing their problems. After 15 
minutes, each group was allowed to watch the video again, stopping the video at any point that 
they wished (both groups only requested to review a few scenes). 

In the following class session, students were given the expert vignettes to read. They were 
then instructed to copy each of their problems onto the categorization form under one or more 
experts and to select which of the experts they would most like to work with. In the third class 
session, students were regrouped around the nine expertise area. The categorization forms were 
cut up and the individual students' problems were distributed to each of the nine expertise groups. 
The groups were instructed to review the vignette of their expert and use the individual students 
problems to create a master set of the most important problems to work on with such a person. On 
the final day, we made a cursory attempt at letting the students in each expertise group try to find 
information about their problems in several pieces of NASA educational materials which we 
provided. 

Resylte and Cgnglusions 

Number of problemSw The 15 students in the generative video condition recorded a total of 
1 12 problems (Table 3). The number of problems per student ranged from 4 to 12, with a mean of 
7-5 problems per student The 14 students in the comparison condition recorded a total of 1 13 
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problems, ranging from 2 to 12 problems, with a mean of 8.1 problems per student Thus there 
was no obvious impact of the type of video on the number of problems students generated 
Content of problems. In order to determine how effectively the videos delimited the 
problem space, we attempted to determine the proportion of problems which were not meaningful 
questions related to planning a Mars Mission. After extensive discussion and several attempts to 
score a test sample, we identified several factors for specifying that a problem was "outside" the 
problem space. This included potential catastrophes (e.g., "What if we run out of fuel?" and 
"What if the rocket blows up?" rather than the inquiry-oriented problems which we were seeking 
such as "How much fuel do we need?" and "What kind of rocket would we use?"). Some 
questions represented reasonable inquiry (e.g., "What does Mars look like?") but were judged too 
far removed from the challenge of planning a mission. Additionally questions where deemed as 
too frivolous (e.g., would we have cable television?"). Blind to the video condition, the authors 
mdependently scored the 225 problems as being inside or outside the problem space The raters 
were in agreement on 76% of the items and disagreements were settled by discussion. A 
proportion of problems rated inside the problem space was calculated for each student The mean 
of this index for the two groups was M = .46, SD = .32, for the generative video and M = .65, 
SD = .22 for the comparison video. While substantial, these differences were not statistically 
significant Efforts to systematically identify the locus of this difference in actual problems in the 
two sets was not successful. 

In an attempt to validate the fmdings of the previous analysis and compare the content of 
the problems from the two groups, the first author categorized the content of each problem 
according to the topic dimensions used to analyze the content of the problems in the first study^. 



^Tbe degree of inference involved in assigning a problem to a particular t(^ic area precluded establishing any sort of 
reliability for this categorization. Because of the low reliability of this measure, we choose not to compute 
prq>ortions of problems inside the problem space for each subject 
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Table 4 lists the frequencies of content stated or implied in each each problem posed by the two 
groups. Collapsing across subjects, 66, or 58%, of the comparison video group's problems were 
"inside" the problem space, compared to 76, or 68% of the generative video group's problems. 
This finding contradicts the previous analysis and points to the continued difficulty . We conclude 
that the more reliable of the two analysis suggests that the problems generated in the comparison 
condition represented a higher level of inquiry, but that the non-significance of the difference 
coupled with the low reliability of the rating and our inability to specifying the locus of the 
difference precludes any conclusive statements regarding the impact of the two videos on the 
delimition of the problems space 

Another question concerned how directly the two videos served to prime students problem 
posing. Examination of the content of the problems posed by the two groups in Table 3 reveals 
that the students in the comparison group posed many more problems which fell under the 
"colonization" and "surface activity" topics, while the generative video students posed more 
problems which fell into the domain of "life on Mars." Examination of these problems suggest that 
these differences reflect different emphases in the two videos. This was an expected finding, and 
suggest that the videos serve to prime problem generation beyond that which would be expected 
from prior knowledge alone. Additional studies comparing video with no video conditions would 
be necessary to confirm the impact of any video on the process. 

Because the comparison video presented a specific mission scenario, we expected that the 
comparison students would generate more questions which make inquiries about specific aspects 
of the information in the video. These are questions or problems related to specific information in 
the video ("What if the plants in the greenhouses die?) rather than other questions which we 
presume to arise more from the learner's knowledge ("Can we grow plants in greenhouses on 
Mars?"). The first author reviewed the questions for both groups, searching for questions that 
appeared to ask specific questions of the information contained in the video. This showed far more 
text- based questions in the comparison- video group (20 out of 1 13) than in the generative video 
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group (3 out of 1 12). However, since it was impossible to evaluate this content blind to condition, 
and most of these questions were generated by just four students in the comparison condition, we 
can't attribute the affect to the video (rather than a misrepresentation of the task). 

Level of inquirv. In order to test whether the generative video condition yielded more high- 
level problems, we ranked problems according to the level of inquiry assigned to the question 
implicit in the problem. Each problem was categorized as yes/no (e.g., "Is everyone in good 
health before they goT\ factual (e.g., "What is the temperature on Mars?), or open-ended (e.g., 
"What kind of rocket would we take?" or "How much fuel would we need?")^. The two authors 
independentiy categorized each problem blind to the video condition. After the initial rating the 
raters agreed on 71% of the items, and disagreements were setUed by discussion. 

The three levels of inquiry (yes/no, factual, and open ended) were assigned the value of 1, 
2 or 3 respectively. A mean score for each subject's problem set was computed. This score 
(minimum of 1 and maximum of 3) was treated as an estimate of the level of inquiry represented by 
the problems each student generated. The mean of this index for the two groups was very sitnilar 
(M = 2. 10, SD = .536 for tiie generative video condition and M = 2. 17, SD = .324 for die 
comparison condition. Thus the expected difference between the two groups favoring the 
generative video was not observed. Once again, the relatively arbitrary scoring criteria, reflected in 
the unsatisfactorily low reliabilities, leads us to qualify the reliability of these results. 

Problem categorization. The new problem categorization activity yielded mixed results. 
The students reported greatiy enjoying assigning their individual problems to different experts, but 
it appeared that many of their problems were too general for them to confidenUy assign them. 
Additionally, some of the students appeared to have difficulty comprehending the difference 
between several of the experts, and during the second phase of the categorization activity, some 



^We iniiiaUy attempted to use a four-level scale with "open-ended" divided into "open-ended with obvious 
paranieters" and "open-ended with unclear parameters" but we were unable to reliably differentiate between the two. 
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students complained that the other stuaents had not provided them with enough "good" problems 
to woiic with. Examination of the final set of "most important" problem in the six categories for 
which we obtained data (Table 5) reveal substantial overlap and miscategorized problems. 

It is not readily apparent whether the use of expert-domain categories is preferable over the 
format used in the first smdy. We changed many aspects of the environment between the first and 
second studies. Furthermore, discussions with teachers revealed that both classes in the first study 
had much more experience with open-ended and unstructured learning activities than the students 
in the study two classroom. Once again, it is not clear whether the difficulty stems primarily from 
limited domain knowledge, poor general reasoning skills, or a misunderstanding of task demands. 
While we viewed the task as an excellent "scientific literacy" activity, the materials and the process 
will need to be refined for this population of students. 

Smdent inquiry. Our cursory attempt at supporting student inquiry was not successful. 
Inadequate support materials and limited classroom time led us to abandon this activity. Most of 
the written support materials were too difficult for these students, while the materials which were 
appropriate to their age level did not contain any of the information they were looking for. While 
some of the students were able to find or deduce information that answered their questions, many 
of the students became visibly frustrated or bored. 

One observation provide a promising view of the apparent level of student investment in the 
problems. Because of the limited support materials and the poorly defined problems, we modified 
several of the expertise groups' problem sets before the final session. We were surprized, but 
ultimately quite pleased by the vehemence of their outrage over our modifying "their" problems 
without their knowledge. While the students agreed that the problems were not useable and 
understood why we made the changes, it appeared that the generation process left them very 
invested in their problems. 

Our experience dramatized the need for more task-appropriate support materials. While 
NASA produces many excellent public-domain print and video materials, most which are made for 
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younger audiences are documentaries and narratives which contain very little technical information. 
This is not surprizing, since these materials are designed to be interesting to a population, which, 
outside of an environment such as our, is not generally interested in the technical content 
Development and testing of support materials is a major long-term goal for our project 

General Discussion 

As could have been expected, these studies failed to conclusively answer our initial 
questions and raised many new questions. In the first study, we were unable to find evidence that 
the the quality of the set of problems improved across the stages of the categorization process. In 
the second study, one analysis showed that the proportion of problems outside the problem space 
was lower for the students who viewed the comparison video. The second study also showed that 
the level of inquiry of the problems for the students who watched the comparison video was almost 
identical to the students who watched the Mars Challenge video. We still lack an adequate 
framework for evaluating student problems and we cannot definitively state which is the most 
effective format for the problem generation activity. We expect this framework to emerge as we 
move beyond the problem generation phase into more formal attempts at supporting students 
inquiry into the problems which they generate. 

We feel that it is appropriate to conclude that these studies demonstrate that our 
environment effectively leads students to generate educationally useful problems. We are 
reassured that despite the problems in these initial trials, students reported enjoying the process, 
appeared to be emotionally invested in the problems which they generated, were able to pose and 
categorize problems in an "educationally meaningful" fashion. Our efforts are guided by the belief 
that a poorly defined problem which the student is emotionally invested in is far more valuable 
than a well-defined problem which the individual has littie intrinsic interest in solving. This is 
particularly important given the oft-cited findings by Miyake and Norman (1979) regarding the 
difficulty of asking good questions in the absence of domain knowledge. Studying the impact of 
the problem generation activities on student interest in both the general domain and in their specific 
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problems is the primary near-term research goal for our activity. Our ideas in this area are 
influenced by an emerging body of primarily European research on content-specific intrinsic 
motivation (e.g., Renninger, Hidi, & Krapp, 1992, Scheifele, 1991). 

Additional research goals include evaluating the contributions of domain knowledge, 
general reasoning ability, task-specific skills, and their interactions, to performance in this 
environment Possible studies include examining how development (grade level) and domain 
knowledge (e.g., high versus average or low, as indicated by student self-report or selected 
populations such as space camp students) hnpacts problem generation. Contrasting baseline for 
performance from older, high knowledge students with younger and/or low knowledge students 
will inform our attempts to optunize each stage of the process. We also hope to devote more effort 
to studying the knowledge which students construct during the generation activity. 
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Table 1 

Content Categories of Problems Posed in Study One 
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Relationship to Category 
Problem Space 



Content Subcategories 



Inside (188) 



Hardware (35) 



Mission (24) 



Health (53) 



Life Support (42) 



Surface Activity (14) 



lUtical (10) 



Fuel/Propulsion (8) 
Construction/Design (7) 
Ventilation/Power (7) 
Capacity (4) 
Communication (3) 
Cs^acity (4) 
Otiier(2) 

Route/Configuration (8) 
Trip Length (7) 
Landing (5) 
Crew Size (3) 
Otiier(l) 

Illness (12) 
Social (10) 
Entertainment (7) 
Toilet (5) 
Cleaning (5) 
Fitness/Training (5) 
Sleep (4) 
Privacy (2) 

Long Term Effects (2) 
aher(l) 

Oxygen (11) 
Food (11) 
Spacesuits (5) 
Water (5) 
Cargo (5) 
Plants/Animals (2) 
aher(3) 

General (5) 
Transportation (4) 
Experiments (2) 
Danger (2) 
Tools (1) 

Cost (6) 
Politics (2) 
(Mer(2) 
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Table 1 (Continued) 

Content Categories of Problems Posed 



Relationship to 
ftoblera Space 



Category 



Content Subcategories 



"Outside (131) 



Mars Conditions (37) 



Colonization (25) 

Life on Mars (14) 
Problems (30) 

Other (7) 

Frivolous (18) 



Atmosphere/Weather (11) 
Temperature (7) 
Gravity (6) 

Volcanoes/Earthquakes (6) 
Geology (7) 

Buildings (5) 
Plants/Animals (5) 
People (5) 
General (10) 

People/Aliens (8) 
Plants/Animals (6) 

Maroon/Lost (4) 
Equipment Failure (11) 
Other (15) 

Unreadable (2) 
Misconception (2) 
What to tell on return (3) 



Generative Video 
27 



Table 2 

Master Categories and "Most Important Problems*' in Study One 



How much oxygen will it take? 
Did you see any people or aliens on Mars? 
How wiD gravity work? 
How much foold and fuel to take? 
Will there be enough technology to get there and come back? 
How long do you stay? 
Is it fiin in space? 

You need a ship that would be able to withstand the heat 

How far and how long wiU it take? They will have to know that to plan how much other stuff 
to bring. 

Feelings About Is it hot or cold? 

Mars Why do they go to Mars and what do they do? 

Is it scary? 

How does it feel in Mars? 
How long will you be there? 
Is this planet foggy? 

How will the surface affect landing? 

Are there any kind of volcanoes, mountains, or landfonns like that? 
Will there be any life? 
Is the gravity going to be greater or less? 
How much space or room will be needed? 
Are there satteliies on Mars? 
Is thexe any water? 

Could we take lifeforms back to earth? 

Safety Is there insurance? 

If something happens to the ship what will they do? 

If you were stuck on Mars how many days will they live? 

If someone got lost would you find them? 

Will there be a doctor or a first aid kit? 

Are there any Mars quakes? 

Are there any volcanic eruptions on Mars? 

What will make a space ship blow up? 

Fuel If they had a solar powered ship it would same a lot of money and fuel. 

They would have to have a lot of gas and chemicals for the going and coming back from Mars? 
Also y ou*d need two sets of l>oosters. One to get out of the atmoshpere of earth and from Mars 
You'd Also need a cooling system for the engines so they wouldn't explode in the extieem heat 
of Mars and its atmosphere? 

And another cooling system in the ship and suits so the men and women that go don't get heat 
stroke, heat exhuastion, or anything else heat related. 

Necessities Do you need to take chemicals up there? 

What kinds of food will you take up there? 
How long will it take to train? 

You*d also have to have to have tools for picking up samples that would melt when picked some 
samples up with it? 

How much extra oxygen will they need? 



Ship & 
Transportation 



Landfoim & 
Lifeforms 
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Tables 

Problem Posing Results in Study Two 



Informational Video (n =14) Generative Video (n = 15 



Total Number Problems 113 1 12 

Mean Number per Student 8.1 7.4 

Problem Categorization (M Problems) 

Yes/No 21 (18.6%) 33 (29.5%) 

Factual 56 (49.5%) 44 (39.3%) 

Open-Ended 36 (31.6%) 35 (31.2%) 

Mean Level of Inquiry 1 2.10 2.17 

Mean Proportion "Inside" Problem Space^ .46 .65 

Number of "Text-based" Problems 20 3 



^ Based on each students mean ratings of problems ranking with yes/no = 1, factual = 2, and open-ended = J. 
^ Based on each students proportion of problems rated as **inside** the problem space. 
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Table 4 

Frequencies of Problems Posed in Specific Content Areas by Students in Informative and 
Generative Video Conditions^ 



Relationship to 
Problem Space 



Category 



Content Subcategories 



Inside (66/75j) 



Hardware (10/75) 



Mission (13/23) 



Health (17/77) 



Ufe Support (19/24) 



Surface Activity (7/2) 



PoUtical ((V7) 



Fuel/Propulsion (3/70) 
Construction/Design (0/7) 
Ventilation/Power (1/0) 
Cj^acity(0/7) 
Communication (2/0) 
Other/General(4/5) 

Route/Configuration (0/2) 
Trip Length (7/*^ 
Takeoff/Landing (0/5) 
Crew Size {214) 
Other/General (4/8) 

niness (1/2) 
Social (1/0) 
Entert^nment (4/2) 
Toilet (1/2) 
Cleaning (2/7) 
Fitness/Training (2/7) 
Sleep (3/3) 
Pregnancy (3/0) 

Oxygen (3/(5) 
Food (9/5) 
Spacesuits (2/5) 
Water (2/2) 
Cargo (1/0) 
Other/General(2/3) 

General (1/7) 
Transportation (2/0) 
Experiments (0/7) 
Danger (1/0) 
What to Return (3/0) 

Cost (0/7) 



Frequencies for Generative condition students given in italics. 



Table 3 (Continued) 



Relationship to 
Problem Space 



Category 



Content Subcategories 



"Outside (47/56) Mars Conditions (9/7) 



Colonization (10/i) 
Life on Mars (6/i4) 
Problems (12/12) 



Atmosphere/Weather (1/1) 
Temperature (5/1) 
Gravity (0/4) 

Volcanoes/Earthquakes (1/0) 
Geology (0/1) 
Look/Size (2/0) 

Buildings (2/0) 
Plants/Animals (5/0) 
General (3/i) 

People/Aliens (2/5) 
Plants/Animals (1/0) 
General (3/9) 

Maroon/Lost (9/7) 
Equipment Failure (2/2) 
Other (1/2) 
Human Failure (0/1) 



Other (10/2) 



Other (10/2) 
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Tables 

Expertise Categories and "Most Important" Problems in Study Two 



Expertise Category 



'Most Important^' Problems 



Psychologist 



Meteorologist 



Chemist 



Astronomer 



Medical Doctor 



Aerospace Engineer 



How would they eat and sleep in space? 
How would the astronauts survive from sickness? 
Do they know what to do in case of an emergency? 
Are there enough oxygen tanks for the suits? 

How would you be able to take a shower in space? 
How much air do you need for space? 
What would they do if they ran out of food? 

How much oxygen would it take to get there and back? 
How much fuel will it take to get there and back? 
If there was a leak what would happen? 
What kind of fuel would you use? 

Will there be any extra oxygen and food if the trip to Mars takes longer 
than planned? 

How long will it take to go and come back? 

Why is Mars the only other planet people can live on? 

How come nobody already lives on Mars? 

Was there ever running water on Mars? 

Was there ever life on Mars? 

What kind of medicines would you use if somebody got really sick? 
If one of the women is going to have a baby, would die atmosphere 
change? 

How do you take a bath in space? 
What if they run out of food? 

How would you keep the Space Craft from burning up when it enters 

Earth's atmosphere? 

How would you get gas if you ran out? 

How would you get water to Mars? 



Mechanical Engineer Data Lost 



Electrical Engineer Data Lost 



Biologist 



Data Lost 
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Appendix A 
Task Instructions for Study Two 

For the next few days, we will be thinking about travelling to Mars. In fact, we are going to help 
you solve some of the problems that will need to be solved before we can go to Mars, We want 
you to think up questions that would have to be answered and problems that might have to be 
solved for women and men to travel to Mars and get back home again. You don 't have to know 
the answers to the questions, and it is okay to list problems which scientists still have not solved 

In a few more days, we will come visit you in your classroom to help you learn 
information that you might use to answer some of the questions and solve some of the problems 
that your class came up with. We will ask some scientists from Vanderbilt to help us figure out 
what we need to learn to solve the problems. So we need to come up with the best questions and 
problems we can think of 

Today you are going to watch a short video. After you watch the video we will ask each of 
you to write down as many questions and problems that you can think of Then we will watch the 
video a second time. We would like to try out two different videos, so one half of you will go to 
one side of the room and the other half to the other side. 
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